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ABSTRACT 

This paper contends that the survey research approach 
is limited for generating substantive data to improve college 
teaching, because it is impersonal, lacks opportunities for probing, 
suppresses the idea of discourse by offering fixed-choice and yes-no 
questions, disregards respondents' social and personal contexts of 
meaning, and is dependent on the competence of coders as ordinary 
language users. In effect, the role of language is ignored in 
collecting faculty evaluation data. Use of the focus group approach 
is presented as an alternative. Three in~class faculty evaluation 
survey instruments were completed by 468 university students, and 
focus group interviews were conducted with 86 students. Survey 
instruments had varying numbers of fixed-choice and/or open-ended 
questions and covered instructor performance, testing, and course 
objectives. Surveys revealed that faculty members were doing a good 
job, leaving unanswered the question of why students were 
dissatisfied with some of their professors and why students were not 
achieving. The focus group approach was more helpful in linking the 
conceptualization of the problem to the collection and analysis of 
data; delineating connections between causes and effects in student 
achievement; and specifying students' need for affection, empathy, or 
belonging to a subculture. (JDDj 
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ABSTRACT 



By examining the results from a number of in-class surveys used for collecting 
faculty evaluation data, this study takes the position that the survey research 
approach is quite limited for generating substantive data to improve teaching. 
The tenor of this study is that because survey research is molded on 
experimental control, quantitative measures, and statistical analysis, the role of 
language has been ignored in collecting faculty evaluation data. Thus, this 
study calls for the use of the focus-group approach, which embodies the view 
that interview is a form of discourse, as an alternative to the survey method. 



INTRODUCTION 

This study is about enhancing the collection of substantive faculty evaluation 
data to improve teaching. The concern over the usefulness of faculty 
evaluations in improving teaching has stimulated intense debate, research, and 
actions at various levels, and has led to the development of programs to address 
the need for improved data collection techniques. Nowhere is the concern more 
significant than at institutions emphasizing teaching. 

While it is true that we must work quickly and vigorously to solve the 
inadequacy of the traditional faculty evaluation approach, it, nevertheless, 
behooves us to place the analysis of this problem on a firm methodological 
footing. Otherwise, the problem will not succumb to rational decision-making. 

Perhaps a "simple folk story" will serve to illustrate the need for skepticism 
in our efforts to delineate those factors that underlie the ever-growing problem 
of collecting substantive faculty evaluation data to improve teaching. 

There was once a man named Nasreddin Hodza who lived in a small, well-knit 
village community somewhere in the Balkan Peninsula. One day he woke up at 
daybreak and went out in the yard in front of his house, bent down on his knees, 
and began a vigorous and meticulous search-inch by inch-through his well- 
kept garden. Soon his neighbors and other passers-by stopped and watched 
him. Curious, they asked him what he was doing. He looked up momentarily 
and said that he had lost his precious gold coin the night before and was trying 
to find it. Now there were not too many such gold coins around the village, and 
the good neighbors, sympathizing with Nasreddin, volunteered to search with 
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him. 

The search continued for hours with increasing., urgency, but the gold coin was 
nowhere to be found. As sundown was approachin^. one frustrated neighbor 
asked Nasreddin, "Now see here! Where approximately do you think you lost 
this famous coin of yours?" Nasreddin looked him straight in the eye and said, 
"I am sure I lost it inside the house. " "Then why in Heaven 's name are we 
searching for it out here?" snapped the exasperated neighbor. Nasreddin shook 
his head knowingly and muttered, "Because there is more light out here!" 

As the preceding excerpt suggests, scholars investigating the problem of 
collecting substantive faculty evaluation data to improve teaching should 
remember that in their pursuit to be 'scientific' they too may be looking for the 
elusive concepts and causes of the problem in areas where there is more 
adequate light for them to conduct the search. 

The present study calls, therefore, for the need of skepticism in our efforts to 
discern those factors that underlie the limitations of the survey approach used 
in faculty evaluations. Subsequently, the rest of this study is divided into five 
major sections: (1) the limitations of in-class surveys for collecting substantive 
faculty evaluation data to improve teaching, (2) the focus-group method, (3) 
results from in-class faculty evaluation surveys, (4) results from focus-group 
interviews, and (5) conclusions. 



THE LIMITATIONS OF IN-CLASS SURVEYS 
FOR COLLECTING SUBSTANTIVE FACULTY 
EVALUATION DATA TO IMPROVE TEACHING 

The in-class questionnaire is seen by institutional researchers as advantageous 
in collecting faculty evaluation data because of its relatively low cost, reduction 
in biasing error, greater anonymity, and considered answers and consultations. 
This type of questionnaire, however, is quite limited for ferreting out many of 
the not-so-obvious answers that would lead a teacher to improve his/her teaching 
skills because it (the questionnaire) is quite impersonal, lack opportunities for 
probing, and limited to influences that propel students toward passing or failing 
a course. Furthermore, in-class surveys do not explain why some students with 
all the characteristics for doing excellent work get lower grades and why those 
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who exhibit none of the characteristics of an excellent student get higher grades. 

Furthermore, the in-class questionnaire, by virtue of being a survey tool, has 
a number of other notable shortcomings. First, in the mainstream tradition rf 
survey research, the idea of discourse is suppressed in the in-class questionnaire. 
Questions and answers, for example, are regarded as analogues of stimuli and 
responses rather than as forms of expression. Thus, the overwhelming majority 
of the questions in an in-class survey are fixed-choice and yes-no types. The 
reason these types of questions are widely used in in-class surveys is to facilitate 
ease in coding. Open-ended questions are said to yield responses that may be 
too complex in intention and wording; such responses in turn make the job of 
the coder very difficult. 

A number of problems emerge with fixed-choice and yes-no questions. 
Encountered with such questions, the respondent will try to interpret whatever 
statement that follows as meaning either yes or no. This does not imply that 
only the forms yes or no can occur; but that whatever occurs is already pre- 
classified as meaning yes or no, or if this is not possible, then as querying or 
refusing assumptions in the questions. The meanings yes or no may be selected 
simultaneously with choices from either systems, including + certainty (Stubbs 
1983:104-105). Yes-no and fixed-choice questions therefore provide a constraint 
in finding coherent answers that can be met by a respondent: no matter what one 
says/writes only confirms or denies a proposition (Schiffrin 1985:643). 

Second, the suppression of discourse in in-class surveys is accompanied by 
an equally pervasive disregard of respondents' social and personal contexts of 
meaning, both in the questionnaire itself, where standardization overrules the 
particularities of individual and setting, and in the modes of interpretive 
theorizing about responses. (The interested reader can be well served by reading 
Milroy 1987.) 

Third, in-class surveys entai! the central problem for coding which may be 
stated as follows: because meaning is contextually grounded, inherently and 
irremediably, coding depends on the competence of coders as ordinary language 
users. Their task is to determine the meaning of an isolated response to an 
isolated question, that is, to code a response that has been stripped of its natural 
social context. Their competence consists in their being able to restore the 
missing varied assumptions they employ as ordinary language users. Although 
coders may share a. common linguistic culture, there is considerable individual 
variation in frames of reference, values, and levels of understanding (Mishler 
1986:3-4). 
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Finally, a related problem appears in the way statistical analyses enter into 
interpretation. Statistical techniques allow testing for significant differences 
across population subgroups, often distinguished from one another along social 
attributes, singly or in combination, such as gender, age, social class, etc. As 
is true of coding, interpretation of these differences relies on unexplicated 
assumptions about the meanings of questions and responses. For example, 
interpreting differences in the frequency among social classes of a particular 
response to a specific question depends on the assumption that the question 
meant the same thing to all respondents. Excluded from this line of reasoning 
is the possibility of variation among subgroups in their understandings of 
questions and in the intentional meanings of answers in how these are, in turn, 
related to variation in socio-cultural frameworks of language and meaning 
(Mishier 1986:5). 

In sum, because the in-class survey is molded on the experimental method, 
which emphasizes standardized procedures, experimental control, quantitative 
measures, and statistical analysis, the role of language has not been given full 
consideration. This situation clearly calls for alternative approaches that will 
delineate the actual underpinnings for improving teaching through the use of 
faculty evaluations. These alternative approaches must embody the view that 
interview is a form of discourse. The contrast between this view and the 
assumptions of mainstream survey interviewing is to be used to develop a 
framework for systematic exposition of the alternative. One such approach that 
bus proved to be quite useful in this vein, and is relatively inexpensive, is the 
focus-group approach used for this study. 



THE FOCUS-GROUP METHOD 

The focus-group method gained impetus from psychology's group dynamics 
approach. The focus-group approach is based upon the notion that talk is a 
relatively cheap from of feedback on everything from popcorn to nuclear bombs. 
It is a variation of the depth interview in which a small number of individuals 
are brought together for interviewing, rather than being interviewed one at a 
time as in the depth interview. The dynamics of the encounter are quite 
different from the individual interview, in which the flow of information is one 
way (from the respondent to the interviewer). The focus-group setting allows 
the comments of each person to be considered in group discussion. 
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Most focus groups consist of eight to twelve members. Smaller groups are 
too easily dominated by one or two participants, while frustration and boredom 
can set in with larger groups as individuals have to wait their turns to respond 
or get involved. (For more on this approach, see, for example, Krueger 1988, 
Stewart and Shamdasani 1990.) 

RESULTS FROM IN-CLASS FACULTY EVALUATION SURVEYS 

The results from three different in-class faculty evaluation survey instruments 
used at three different universities were examined. The first survey 
questionnaire comprised ten (62%) fixed-choice and six (38%) open-ended 
questions; the second had fourteen (78%) fixed-choice and four (22%) open- 
ended questions; the third had thirty nine (100%) fixed-choice questions. The 
fixed-choice questions of all three survey instruments were to be answered using 
a scale from five to one, where five means "strongly agree" and one means 
"strongly disagree." 

The survey questionnaires examined were completed by 468 students. More 
than four-fifths of those students who completed questionnaires with open-ended 
questions did not bother to answer those questions. Of those who provided 
answers to ?uch questions, many gave one- word responses such as "None." The 
surveys generally had three parts: (1) instructor performance, (2) testing, and 
(3) course objectives. 

The results from one of the survey instruments comprised means and standard 
deviations for the factors identified by factor analysis of the questions. The 
questions comprising each factor were indicated in order of factor loading. 
Means were based upon the total responses within each of the indicated 
summary levels. The professor's means were then compared to those of his 
unit, his university, and a national sample of 200,000. 

For the other two survey instruments, the average scores and percentages of 
the responses to each question were provided. These statistics were then 
reported side-by-side those of the professor's unit. 

An analysis of the results from the computer runs for the 468 completed 
questionnaires can be summarized as follows: Out of possible means of 5.0, (a) 
the mean for instructor performance was 4.03, (b) the mean for testing was 4.01, 
and (c) the mean for course objectives was 4.07. In sum, these scores indicate 
that respondents generally perceived faculty members at all three universities 
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examined to be doing a good job. This finding leaves unanswered, however, 
why so many students were dissatisfied with some of their professors and why 
a number of students were not being successful in their courses as revealed in 
the focus group interviews. 

Furthermore, a faculty evaluation report typically outlined three general 
purposes for the evaluation: (1) to help the professor improve his/her teaching 
ability by pinpointing potential weaknesses; (2) to forward to the appropriate 
committees which make decisions on course offerings, faculty hiring, 
promotions, and tenure; (3) to enable future students to make judgments on 
whether or not to take the course or the professor. Here again, an important 
question emerges: How will faculty evaluation reports which present only 
means direct a professor on how to improve his teaching strategies? 



RESULTS FROM FOCUS-GROUP INTERVIEWS 

Eight focus-group (each group comprised from nine to twelve students) 
interviews, involving eighty six students, were conducted for this study. Each 
of the groups was led by two faculty members. The faculty members followed 
a list of questions as a discussion guide, asked appropriate follow-up questions, 
prevented any one participant from dominating the group, and ensured that every 
member participated. The results that follow are based on discussions with the 
eighty six students who participated in the focus groups. Thus, it is important 
to note that no attempt is made to generalize from these results. 

None of the results discussed in this section, which are neither exhaustive nor 
mentioned in any particular order, would surprise professionals working in 
academia. Throughout the focus-group data collected, there were several 
recurring themes. 

Students particularly liked professors who (a) outlined key points, (b) covered 
just enough material in detail, (c) compared present developments with those of 
the past, (d) encouraged a relaxed atmosphere, (e) provided feedback in a timely 
manner, (f) used a mixture of visual, auditory, and kinesthetic teaching 
strategies, and (g) displayed "affection" or "empathy." Most of the students' 
comments, however, were about evaluating student performances and the 
learning environment. 
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Evaluating Student Performances 

Most students suggested that in order to effectively evaluate their cognitive, 
metacognitive, and social affective skills, which are crucial to the successful 
learner, a mixture of in-class examinations and take-home assignments is called 
for. They noted that the following presuppositions seem to support their 
suggestion: 

1. Examinations are to acquaint those unfamiliar with time limit and 
written directions but without disaster, and take-homes are not timed. 
Grades, individual notes, and comparing the comments should give a better 
idea of what is expected. 

2. Questions presuppose facts involved are familiar and so set a framework 
for discussion. Many questions have no one solution, and sometimes the 
'best' answers are those with no single answer everyone would accept. 

3. The lectures presuppose students have studied the texts beforehand and 
read other sources referred to, just as tests presuppose they have thought 
about the topics and discussed them with others. It is taken for granted 
that the students' initial, sole, and final acquaintance with the subject 
matter is not just the lectures. 

4. Take-homes are to help, not annoy. They are done at students' pace, 
with any kind of aids they find helpful. They are to be taken as learning 
opportunities, not just chores. Cooperation is to be indicated: a single 
student should not have to answer every question, but group-members can 
cover the lot. Hardy workers can take on questions where the answer is 
not obvious, since that is how people learn. Comparing results helps; 
preparing for likely questions helps even more. 

5. Every student has his/her personal ways to study and review, but it is 
sensible to go review what s/he has spent time on, by keeping, reading, 
and revising what s/he has written. If one's work is not worth rereading 
and reworking, it was not worth doing in the first place. Knowledge is 
cumulative—a student should be his/her own severest critic. Regarding 
assignments as something to be forgotten when handed in wastes a 
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student's time. Rewriting isolates essentials, shows what is peripheral, and 
which writing habits waste time. Supplementary reading illuminates what 
was obscure, provides better examples, and brings up references a student 
could not consult when s/he first discussed it with others. 

6. Cooperative reading can be even more profitable than cooperative study: 
an individual reports in minutes what took hours to read, tell others what 
s/he thinks is not worth reading, or what they just have to read for 
themselves. Setting brief but regular times for informal reports about 
supplementary readings multiplies a student's reading-there is only so 
much one student can do alone. 

The essence of these presuppositions is that to foster some kind of discipline, 
an obligatory take-home format forces students to make their own work re- 
identifiable, reliable, and worth rereading. The take-home assignments then 
constitute a text for a student's final examination review, future study, and 
updating. Duplicating and exchanging results with, others makes them complete. 

The Learning Environment 

In talking with students who participated in the focus groups, it was 
discovered that these students clearly desired to perform well in their courses. 
This finding was not unrelated to those of other scholars who talk about the 
'poor student performance syndrome.' Yet until the focus-group discussions 
were held and students talked about their own personal situations, it was not 
realized that the rate of poor performance among some students was the desire 
and need for "empathy 11 and "affection" that were not forthcoming from a 
number of professors. A matter of fact, some students who were performing 
poorly were giving so much "love" at home by taking care of grandparents, 
parents, siblings, spouses, etc. that they received relatively little "love" in return. 
Some saw the university as a domain in which they could receive at least some 
"empathy," if not "affection." 

"Students caring about other students," as some students frequently stated, 
ranked the highest on the lists of focus-group participants, and explained more 
of the variance in students' expectations of faculty members. This situation 
clearly highlights the difficulty of employing any type of analysis that hinges on 
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in-class faculty evaluation surveys in conceptualizing basic explanatory 
variables. 

Another important finding from the focus-group discussions was that the 
larger university culture displays a lower degree of integration with a number 
of elements that do not fit well together. Thus, there exist "subcultures" which 
are distinctive variations of the larger university culture: a professional attitude 
in the pursuit of higher education. Focus-group participants described several 
types of "subcultures," each one with its own set of behavior patterns operating 
at universities. Consequently, some students become confused and frustrated 
when faced with the situation of having to determine exactly where they fit. 
Sometimes, the behavior of a "subculture" runs directly counter to those of the 
larger university culture. 

One of these "subcultures" can be referred to as a "classroom subculture." 
Almost all interviewees suggest that there exist beliefs and behavior patterns in 
some classrooms that differ with those of others. The "subculture" of some 
classrooms was described as "very professional and businesslike." Respondents 
also noted that because some classrooms were new and cleaner, they conjure up 
an atmosphere of "professionalism": professors and students treat each other 
with "dignity and respect." Some classrooms, on the other hand, were described 
by respondents as having a "high school atmosphere." Students pointed out 
untidy surroundings and "unprofessional" attitudes of many professors and 
students as attributes endemic of a "high school environment." 

Also prevalent at universities is what can be called a "group subculture": a 
set of students who interact with one another and share some joint activities. A 
frequently mentioned "group culture" is the athletic one. A number of 
interviewees spoke of athletes, cheerleaders, and physical education professors 
belonging in an "exclusive club." While two athletes and a cheerleader in one 
of the focus groups spoke highly of the camaraderie between athletes, 
cheerleaders, and physical education professors being one of their positive 
experiences at the university, other participants complained that the group 
behavior of this "exclusive club" sometimes becomes counter-productive. One 
interviewee put it this way: "If you are a cheerleader or an athlete in one of the 
physical education classes, you can get away with murder." Another interviewee 
stated that in one of her physical education classes, athletes hardly did their 
work, clowned during lectures, and still ended up with good grades. 

The lack of information was mentioned several times as a problem 
encountered by participants. One student on a university's basketball team 
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expressed his surprise at the amount of work required for a typical course. He 
reported that his schedule and course load were too heavy, given his athletics, 
even though his schedule had been put together by his faculty advisor. He said 
that had he known the amount of work required for his course load, he would 
have taken fewer courses. Whether he had been told that a three-credit course 
typically requires a minimum of nine hours of work per week is, perhaps, 
irrelevant; the point is that he did not know this rule of university life. Such 
basic information might have been passed on through an orientation session, but 
few of the participants recalled attending one. 

Students expressed their surprise at the cost of books. One participant had 
a scholarship that covered her tuition. Coming from a public school system 
where books tend to be borrowed, rather than purchased, and uninformed of the 
cost of books required for her four courses, she was unprepared for the financial 
difficulties encountered the first week of her university career. Another 
complained about the need to purchase several textbooks for a given course 
when only one was actually used by the professor. The changing of textbooks 
from semester to semester was also mentioned as a problem, since this practice 
reduced the number of used textbooks that could be purchased and used again. 
A participant recalled how he had fallen behind in course-work because he could 
not afford a textbook during the first few weeks of the semester. 

What was intriguing was that a number of interviewees admitted that they 
were academically ill-prepared for university life. But they were also quick to 
point out that not having faculty advisors when they needed them, or having to 
talk to a different faculty advisor each time, did not help their situation. Some 
also noted that certain professors found it burdensome when students requested 
additional help with their assignments. One interviewee recalled a professor 
saying: "I've got mine, you need yours. If you want it badly enough, you 
would have to get it yourself. I am not here to baby any student." On the other 
hand, one student spoke most highly of one of her professors who caringly 
chided her for any tardiness, got students to call her when absent, and-by 
saying the right things at the right times-taught her the importance of 
promptness. 

To summarize, the focus groups provided additional inforniation related to 
improving teaching. "Facts" suggested through the cold analysis of dry statistics 
dealing with in-class faculty evaluation survey data can be fleshed out by further 
dttails in discussion groups focusing upon specific topics. The additional 
information supports, to name a few recommendations, the need for setting up 
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a more supportive learning environment, improving the dissemination of basic 
information for academic survival, devising ways to reduce the cost of books, 
and training faculty members in methods that enhance, rather than diminish, 
student success. 



CONCLUSIONS 

Both careful speculation and informal data collection were needed in using the 
focus-group approach in this study. In order to see enough of the whole picture, 
adequate conceptualization of the major explanatory variables underlying the 
problem of using in-class faculty evaluation surveys to collect substantive data 
to improve teaching was required. It also involved operationalizing certain 
concepts in ways that adequately represented "affection," "empathy," and 
"subcultures" in diverse communities of students, determining the complex 
linkages between the variables examined, and deriving the basic propositions 
tested. It is this stage where researchers on faculty evaluation find or construct 
empirical indicators and proceed. This is why they miss important elements in 
their investigations. 

A missing element in a number of faculty evaluation surveys is coherence: 
the issue of using results gleaned from these surveys to help improve teaching 
is often poorly conceptualized and then data are collected that have little or 
nothing to do with resolving the problem. The requirements for focus-group 
interviews, on the other hand, helped in the initial conceptualization of the 
problem and linking the conceptualization to the collection and analysis of the 
data. 

Another way the focus-group approach became very useful was in the 
specification of certain relationships between ideas. We were able to delineate 
logical connections between causes and effects on the student success problem 
that made it possible for us to see clearly what we were talking about and how 
we were thinking. After linking together pertinent ideas from our focus-group 
interviews, we were able to generate deductions from them. This forced us to 
specify relationships for the concepts of "affection," "empathy," and 
"subcultures," which our ordinary usage of the English language normally slides 
over. Once we grasped the underlying factors for the student success problem, 
we were able to specify relationships between what students expected and their 
need for "affection," "empathy," or belonging to a "subculture" in ways that 
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otherwise would have been merely assumed in a survey study. 

In addition, the focus-group interviews helped us make deductions from ideas 
we otherwise would not have thought about. This is one reason it became 
evident, throughout the data analysis, that a number of our findings ran against 
standard findings of scholars using "behaviorist" methods is studying the student 
success problem in higher education. 

Indeed, the focus-group approach helped us pinpoint what our investigation 
was and what we needed to ferret out. They provided us with a way of thinking 
that helped us through the problem of understanding what a university meant to 
the students interviewed and what professors and students need to do in order 
to combat the ever-growing student success problem. This was no small feat! 

By recognizing the centrality of language in our research, we were able to 
demonstrate that language is the most significant and colossal work that can help 
us grapple with the problem of student success in higher education. But because 
language is one of the most characteristic forms of human behavior, it is so 
familiar a feature, of our daily life that many researchers rarely pause to think 
that it is the source for our answers to the ever-growing student success problem 
in higher education. 
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